1. Field of the Invention
The present invention concerns an information processing system including an adaptive, sensory-motor encoder for a visual prosthesis or acoustic prosthesis for bi-directional coupling using implanted micro-contacts both for stimulation of neural or glial tissue as well as for the purpose of functional monitoring of brain function.
2. Description of Related Art
A number of attempts have been made to develop vision prostheses for various groups of blind persons by implantation of micro-contacts at the output layer of the retina (RET) or within the visual cortex (VCO) and by coupling these implants with an external signal transmitter (the encoder) in order to elicit functional visual perceptions. For example, encoders for implantable vision prostheses are described in U.S. Pat. No. 5,498,521, U.S. Pat. No. 5,351,314 or WO95/06288; implantable micro-contacts for the retina, visual cortex or auditory system are described in U.S. Pat. No. 5,597,381; U.S. Pat. No. 5,569,307; U.S. Pat. No. 5,549,658; U.S. Pat. No. 5,545,219; U.S. Pat. No. 5,496,369; U.S. Pat. No. 5,411,540; U.S. Pat. No. 2,215,088; U.S. Pat. No. 5,109,844 and U.S. Pat. No. 4,628,933. U.S. Pat. No. 5,277,886, EP 0435559 and U.S. Pat. No. 3,766,311 are concerned with neural networks and the visual system and U.S. Pat. No. 5,571,148, U.S. Pat. No. 5,512,906, U.S. Pat. No. 5,501,703, U.S. Pat. No. 5,441,532, U.S. Pat. No. 5,411,540 are concerned with addressing micro-contacts.
The target groups of the RET projects suffer retinal degenerative disease (for example, retinitis pigmentosa, macular degeneration) whereby the photoreceptor layer has degenerated but at least a portion of the retinal ganglion cells and part of the optic nerve originating there, as well as the central visual system are still functional. As is demonstrated by the publications mentioned above, work is being done on development of a variety of types of implantable micro-contact structures (stimulators), that are applied inside the globus oculi (eyeball) onto the ganglion cell layer of the retina and on the development of wireless signal and energy transfer systems for connection between the external encoder and the implanted stimulator, or generally to the interface.
The inventor has confronted the task of further developing an adaptive encoder for a visual prostheses that is coupled at the retina or to the visual cortex for conversion of image patterns or for acoustic prostheses coupled to the appropriate areas of the neural auditory system for conversion of sound patterns into stimulation signals through adaptive, spatio-temporal filters using receptive field characteristics of the respective sensory neurons (RF filters) addressed and their optimal adjustment by neural networks acting as adaptive function approximators.
The target groups of VCO projects typically no longer have recoverable optic nerve function and therefore require implantation of, for example, comblike micro-contact structures into regions of the visual cortex; that is the occipital cortex, that are directly adjacent to the cranium.
There are also several familiar types of acoustic prostheses (for example, the cochlear implant) using implanted micro-contacts that make possible partial recovery of auditory perception in deaf persons.
The current designs of implant systems do not provide for any information processing of the dynamic image pattern data within the visual system (for example, the retina), from the optical input up to the neural layer contacted by the implanted micro-contacts (for example, the ganglion layer of the retina or the neural layer in the visual cortex). Instead, simple image patterns (for example, lines) are forwarded directly as stimulation signals at the locally distributed micro-contacts without individually adapted information processing as a substitute for that part of the visual system that is to be technologically bridged (for example, the retina). The visual system that has been so rudimentarily and unadaptively stimulated is confronted with the difficult problem of generating visual perceptions, that are sufficiently similar to the image pattern, from the locally and temporally incorrectly processed or coded signal paths. Furthermore, the physiological adjustment of the visual sensitivity (brightness adaptation) over approximately 10 decades and the functional alteration of the receptive field characteristics or features connected with it in technical photosensor systems is not taken into account.
The currently developed implant systems using available technology do not use visual prostheses for the purpose of warning the implant carrier of hazards and reporting technically identified patterns. The same applies to currently developed implant systems for the auditory system.
Because of its ontogenetically established structure and its stabilized structure and function through years of visual perceptual experience prior to the occurrence of blindness, the visual system expects, for example, a particular kind of information processing of the image pattern data by each retinal ganglion cell via the optic nerve. This expectation is neurophysiologically transcribed by the corresponding receptive field features (RF) of the neuron and is very variable, is not fulfilled, inasmuch as, for example, the function of the retina, electrically stimulated by static pre-processing, cannot be individually adjusted for each single, stimulation contact produced by implantation. The same applies to the auditory system.
The collaboration necessary for the normal vision process of the central vision system receiving its inputs from the retina with the respective eye movement system is called the sensory-motor xe2x80x9cActive Visionxe2x80x9d system.
Such systems for pattern recognition, object tracking, object position recognition, etc. are taken into consideration in conventional applications. The visual system nevertheless requires eye movements for all vision performances and produces not only recognition performances (what?), but very importantly recognition performances (where?) as well as orientation in space, all of which have a very high priority for the visually challenged person who is in the process of partially recovering his ability to see. Triggering of visually induced eye movements, however, in the case of actual stimulation by the use of the implant system, cannot be expected from only a small fraction of the retinal ganglion cells or the cells in the visual cortex. Therefore, the visual system, using the implants that are currently in the development stage and which are based on normal cooperation with eye movements, can perform only unsatisfactory visual perception, if any at all.
In various visually challenged persons there occur, in addition, undesired, non-visually induced eye movements with slow and fast phases, that can significantly impair optimal utilizationxe2x80x94and thus the acceptancexe2x80x94of this type of visual prosthesis. If, for example, the encoder with photosensor array is fixed onto the globus oculi, the then desired locus of fixation is constantly shifted by the undesired eye movements. If, on the other hand, the encoder is built into an eyeglass frame, then the visual system will interpret the image pattern with the eye movements that have not been harmonized with the image pattern, as ambiguous visual perceptions, for example, as apparent movement, as is, for example, the case in vertiginous perceptions.
Without the possibility of visually induced, real eye movements and the additional conflict with undesired spontaneous eye movements, visual orientation in space, position identification of various objects relative to the location of one""s own bodyxe2x80x94for example for the intentional grasping of a door knobxe2x80x94using visual prostheses currently in development, which are dependent on head and upper-body movements for the change in direction of vision, are barely possible. The structures to be implanted have a very limited number of micro-contacts. The number of effective usable micro-contacts is even smaller, since only a fraction of the contacts can be positioned by implantation relative to a nerve cell or fiber, so that with the use of individual contacts, or contact pairs, neural action potentials with acceptably low stimulation amplitudes can be triggered. In the cases of current development, there is hardly the opportunity to increase the number of permanently and selectively contacted neurons beyond the quantity accidentally established at implantation. This is one reason for the only minimum visual perception quality that can be expected. The same applies to implant systems in the auditory system.
The currently developed micro-contacts and signal and energy transfer systems for implantation of visual prostheses function unidirectionally from the external encoder to the implanted stimulator and therefore offer no opportunity for ongoing monitoring of the neural impulse activity of the stimulated neurons. Thus, the stimulation pulse frequency cannot be adjusted to the spontaneous activity of the neurons. Furthermore, the triggering of neurobiological impulses by stimulation (excitatory) pulse cannot be directly monitored. Furthermore, there is no sure opportunity for impulse monitoring for a possible temporal tuning and synchronization of the impulse sequence of several neurons. The same applies to the auditory system.
The task of the present invention is to eliminate these disadvantages and to provide an adaptive sensory-motor encoder.
This problem is solved by a device having the characteristics described in this disclosure.
Because the encoder operates in bi-directional coupling with implanted micro-contacts on the retina or on the visual cortex afterxe2x80x94preferably with the aid of neural networks in carried out in dialogue with the implant carrierxe2x80x94functional tuning as individually required, various visual recognition, tracking, and location identification tasks as well as reporting of hazards and technically identified patterns can be performed by technical image pattern shifting and simulated eye movements, the number of selectively reachable stimulation sites functionally increased, and the neural activity of neurons to be stimulated can be monitored. Furthermore, the encoder functions of brightness adaptation and composition of a visual operating range from excerpts of a large function range described here can be done.
With implementation of an acoustic prosthesis, the adaptive encoder can provide corresponding services in the auditory area.
In the case of the preferred design form of the encoder for a visual prosthesis, a digital signal processor (DSP), for example, the Texas Instruments model C80, is integrated with a photosensor array with optics as the light pattern receiver, a pre-processing module for visual patterns, an impulse signal emitter and receiver for bi-directional communication with the implanted structure, several signal interfaces for communication with the evaluation input unit, the head movement sensor, the eye movement sensor, the perception, warning, and recognition system, pattern and object reporting system and the external monitoring and control system integrated into an eyeglass frame. The various adaptive information processing functions, particularly for RF filters, dialogue module, pattern recognition, and Active Vision functions are provided in the DSP with a central control unit. The user receives, on the one hand, signals as stimulation impulses or receives sensory perceptions from the encoder and transmits, on the other hand, signals regarding head and eye movements as well as the evaluation entry and neural activity to the encoder. Because of a bi-directional wireless signal and energy transfer, the encoder can be installed in an eyeglass frame, attached to a contact lens on the eye, placed on the body, or located at a body-remote site. Finally, the spatio-temporal functional range of the RF filter used as a retinal encoder includes, for example, the receptive field characteristics of retinal ganglion cells or other intra-retinal neuron classes of the primate retina. The same applies to the preferred designs of the encoder in coupling with the neurons of the visual cortex or in the case of an acoustic prosthesis for coupling with neurons of the auditory system.
A suitable form of a procedure for adjustment or setting the RF filter of the encoder in the dialogue with the user is illustrated schematically in FIG. 1 and FIG. 2. The RF filters are executed as a spatio-temporal filters whose spatial and temporal functional parameters are modified in a sufficiently large function range for approximation of the receptive field characteristics of visual neurons, namely, by externally accessible parameter exchange points placed at appropriate positions in the filter algorithms. A human being communicates, as a normal-sighted person or implant carrier in a perception-based dialogue with the encoder, the perception comparisons between the desired and the actual patterns, for example, through an evaluation input unit composed of a line of several dip switches (see FIG. 2), to a technical neural network with non-monitored adaptation rules. The neural network then establishes the next parameter vector for the RF filter as well as the next desired pattern, with the goal of reduction of the perceived pattern difference in the next dialogue step. In the search for optimum parameter vectors for the RF filter, parameter vectors that result in a certain visual perception for a given light pattern presented and appropriately subjectively interpreted can be produced in the dialogue module of a neural network using non-monitored adaptation. Alternatively, in the dialogue module, another parameter-setting system can produce sequences of parameter vectors for virtual movement in the functional space of the RF filter, for example, as continuous trajectories depending on the type of scanning or sweep, as rule-less sequences or as sequences of neurophysiological especially typical filter functions, and the user,during this sequence running in an appropriate timing scheme, occasionally reports xe2x80x9csensiblexe2x80x9d perceptions resulting from the interaction of the given light pattern, the series connected pre-processing module, the series connected RF filter and the part of the central visual system coupled via the appropriate micro-contact. Then, a more precise perception-based parameter optimization is carried out in the so-determined range of the filter function space. A suitable form of the dialogue-based setting of the RF filter of the encoder of an acoustical prosthesis is done in similar fashion.
In the generation of asynchronous impulse sequences, the output signals of the particular RF filter are, through suitable conversion algorithms of the quasi-continuous time functions of the RF filter, converted to asynchronous impulse sequences suitable to the activity of visual neurons in the primate visual system, and impulse sequences-time courses, as well as onset times of a particular impulse, are adjusted through variable time delay elements during the dialogue phase. A suitable form of the vision system model for the perception comparison in normal-sighted persons is found in the fact that for a series of RF filters, individually or as a group, the respective appropriate inverse representation is provided and thus parameter vectors are established as precise. Through sequential switching with the encoder an actual image is produced on the right screen with considerable similarity to the desired pattern. At the start of the dialogue the parameter vectors of the RF filter are set to random start values so that initially there is a clear difference between the patterns, but in the course of the dialogue with non-monitored adaptation becomes consistently less.
The functional adaptation of the RF filters for the implant carrier in the perception-based dialogue occurs in contrast with the functional adaptation for normal-sighted persons in that the actual perception is not accessible on a monitor but is only internally accessible to the implant carrier and that the desired perception is communicated to the implant carrier is, for example, as speech information or as a tactile contact pattern on the skin. When used in acoustic prostheses a similar situation exists wherein, instead of the inaccessible auditory organ, for example, along with the tactile sense the available vision sense can be employed to communicate the desired perception.
The temporal coupling of the asynchronous impulse sequences produced by several RF filters of the encoder for triggering of neural impulses occurs in that the transmission time points of the individual impulse signals are varied by controllable time delay elements such that there is a temporal coupling resulting in precisely synchronous occurrence so that the variation of the time delay is controlled by the implant carrier, results in the dialogue as perception-based event by way of a neural network or is controlled externally, that the selection of the impulse groups to be coupled temporally can be taken into consideration [berxc3xccksichtigt] both by the impulses coming from the RF filter as well as the impulses recorded in the interface, and that in view of the very different momentary impulse rates of the various RF filters suitable criteria are established for the participation of individual impulses in the impulse groups to be coupled.
For the purpose of functional enhancement of the number and of the selectively reachable stimulation sites with a given number of stationary micro-contact implants, the impulse signals from a given RF filter are guided to several, neighboring micro-contacts.
The characteristic time courses of the electromagnetic field in the area of the neurons to be stimulatedxe2x80x94based on the encoder commands and set exactly for each micro-contact and decoded stimulation time functions corresponding in the interface with respect to current amplitude, polarity and phase lengthxe2x80x94have the effect that these stimulation signals that are tuned to each other by superpositioning, trigger local and temporally selective neural impulse excitations a field strengths at several micro-contacts. The selective simulation occurs through appropriate variation of the superimposed stimulation signals and can be rapidly changed. The corresponding variation of diverse parameters of the reciprocally-tuned stimulation signals in the perception-based dialogue with the implant carrier occur via the neural network, or other signal variation processes at the adjacent electrodes, such as, for example, a continuous, sweeplike automatic shift of the functions parameters of the stimulation impulses that are superimposed upon the nerve tissue and used in the determination of as many as possible stimulation sites that will result in neural excitation, In addition, through the comparison of recorded neural impulses with the stimulation signals, the optimization of the stimulation time functions with respect to intended single-cell selectivity and permanent biocompatibility is improved.
For the purpose of simulation of eye movements for utilization of the encoder in a vision prosthesis, an image pattern shift is done electronically in the input layer of the encoder by optical variation of the direction of vision, for example, with the aid of a moving mirror or by movement of the photosensors for simulation of eye movements. Head movements and eye movements are detected by microminiaturized movement detectors operating multidimensionally, and neural or conventional movement control is provided by using the detected head and eye movement signals and the image pattern shift. Rapid and slow eye movements for the tasks of pattern recognition, rapid peripheral scanning or eye tracking movements are produced, and by the appropriate eye movement sequences with fast and slow phases optimal adaptation of the sensory data flow results at the responsive central vision system. With respect to the production of eye tracking movements, there is a suitable design form in that an adaptive neural predictor picks up the initially unknown movement time function of the object to be followed from the position and movement errors of its projection on the photosensor array, and using an appropriate non-linear adaptive prediction algorithm with consideration of the analyzed frequency segments of the object movement, an object tracking-time function with minimal delay or even with minimal lead is generated internally with rapidly increasing reliability and, in the case of temporary disappearance of the object being tracked, for example, similar to the situation in which the eye tracking movement system in primates produces a movement time function, which, depending on the nature of the object, results in a continuation of the tracking on reappearance of the object, which has moved relative to the sensor array, with minimal positional and movement error.
For the purpose of detection of eye and head movements and for compensation of undesired eye movements fast and slow eye movements are produced by using the detected head and eye movement signals and the simulated eye movements [created] with the help of a neural or conventional movement control or guidance. With the aid of the control loop undesired eye movements are compensated following an appropriate period of adaptation and so that an adequately satisfactory simulation of the vestibular-ocular reflex (that is, the automatic reflex stabilization of the direction of vision in space by eye movements that counteract head movements that may occur) is produced from the head movement detector, image pattern shift and a neural network in a control loop optimized in the period of adaptation, allows positional stabilization of the image pattern, in the presence of natural head and upper-body movements, by corresponding compensatory eye movements.
Simulated eye movements and compensatory eye movements are available as separately selected programs. The individual movement and shift modes can be selected as separate or combination programs, allocated to automatic operation or established externally.
The implant carrier can select the desired xe2x80x9cActive Visionxe2x80x9d functions, such as looking around or object tracking, by means of a portable command input device; for example, a handheld device with keypad.
Ongoing communication to the implant carrier of the current position of objects picked up visually or acoustically in space consists of the encoder""s assessment of the object""s position by evaluation of image patterns or sound patterns, eye and head movements using an appropriate, portable signal transmitter reported to the appropriate sensory organ; for example to the tactile sense, and that the encoder by means of an internal pattern recognition program, in particular in conjunction with automatically occurring eye movements, warns the implant carrier of obstacles or hazards and reports type and position of technically identified patterns or objects.
A monitoring system for a partially sensory-motor autonomously operating implanted structure of the encoder consists in the implanted micro-contacts being used both for the stimulation and for recording of neural impulses, such that the recorded impulses and other physical or chemical signals from the implanted structure are reported to the encoder by the appropriate pre-amplifiers and optical or electromagnetic transmitters and that, once there, the recorded neural signals are further processed for the various purposes of the encoder functions.
For the purpose of technical adaptation of the brightness-operating range from a, for example, via a photosensor array function range extending over six to ten brightness levels, that includes both large portions of the scotoptic range of dark adaptation as well as the photoptic range of bright adaptation, a rapidly adjustable electronic display on an internal operating range for the encoder, relative to size and adaptation brightness, is adjusted perception-based, automatically selected or by the user. By doing so, visual scenes of very different brightness are compensated into the current operating range and thus contrast optimization and avoidance of glare is achieved. Furthermore, an advantageous design is inherent here in that the RF filter functions; for example, corresponding neurobiologically familiar processes can be adjusted using the adaptation range.
In a suitable form, a pre-processing module is provided that is equipped for recognition of characteristic patterns and/or for data reduction. When this is done the pre-processing module can recognize bright and dark image areas of differing luminance, separate them from one another, and integrate them to an overall image with the respective regional optimal contrast; likewise image areas lying proximately or remotely separate from one another can be optimized with respect to sharpness adjustment and then reintegrated as an overall image and, finally, characteristic patterns like, for example, warning signs, can be emphasized. This pre-processing can be used to improve the image presentation, but it can be used also for the production of warnings and for data reduction in the communications channel between the camera and the encoder.
It is of further advantage if in the course of a process the adaptation of the accommodation initially, for example, in the foreground, is selectively adjusted, these ranges are stored, and then secondary ranges of the visual field are adjusted sharp as images; for example, the background of a visual field. The initial ranges that now become flat (i.e., unsharp) are faded out of the second image pattern and replaced by the first stored, sharp ranges. By doing this, a depth of definition is produced that is unattainable in the scope of geometric optics. With the cyclically repetitive course of this process step and corresponding image balancing the user remains unaware of the process so that even in minimal light intensity and long focal distances the appropriate optics an image of great definition is apparently obtained.
The adaptive pre-processing module processes the image pattern or sound pattern by neural networks or in pre-processing algorithms for a visual prosthesis accessible in the hardware particularly with respect to color, contrast, edge detection, segmentation, and figure-to-background separation or, in the case of an acoustic prosthesis, for example, with respect to suppression of interference noise, undesired formants and separation of individual sound sources in such a manner that the subsequent RF filter array is considerably simplified and contains imgage patters or sound patterns formed in part from the pattern area in a characteristic or properties area, that is as well adapted as possible to the part of the visual system or auditory system contacted. The various pre-processing functions are set directly by the user or selected by a perception-based dialogue or set automatically. The image simplified in this process and communicated to the retinal encoder is advantageous for figure recognition even in the case of a limited number of visual system neurons contacted. In the pre-processing for acoustic prostheses, corresponding benefits result if complex sound patterns are prepared in the course of pre-processing for speech recognition.